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(57) ABSTRACT 

System and method for optimization of a design associated 
with a response function, using a hybrid neural net and 
support vector machine (NN/SVM) analysis to minimize or 
maximize an objective function, optionally subject to one or 
more constraints. As a first example, the NN/SVM analysis 
is applied iteratively to design of an aerodynamic compo- 
nent, such as an airfoil shape, where the objective function 
measures deviation from a target pressure distribution on the 
perimeter of the aerodynamic component. As a second 
example, the NN/SVM analysis is applied to data classifi- 
cation of a sequence of data points in a multidimensional 
space. The NN/SVM analysis is also applied to data regres- 
sion. 

11 Claims, 14 Drawing Sheets 
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Initialize connection weights of 
neural layer of the NN/SVM shown 
in Figure 2 (random weights) ^ 


Compute outputs of the hidden 
layer; representing coordinate 
directions in feature space 


If necessary, provide 
user-specified feature 
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corresponding inner 
products and/or kernel 
functions 


Compute necessary inner 
products (kernel function) 
for the SVMcomponent 


Compute the Lagrange multipliers 
for the SVM component 
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connection weight 
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HYBRID NEURAL NETWORK AND 
SUPPORT VECTOR MACHINE METHOD 
FOR OPTIMIZATION 

ORIGIN OF THE INVENTION 

The invention disclosed herein was made by an employee 
of the United Stales Government and may be manufactured 
and used by or for the Government for governmental pur- 
poses without payment of any royalties for such manufac- 
ture and use. 

FIELD OF THE INVENTION 

This invention relates to design optimization, using a 
hybrid neural network and support vector machine approach 
to construct a response surface that models a selected 
objective function. 

BACKGROUND OF THE INVENTION 

Considerable advances have been made in the past two 
decades in developing advanced techniques for numerical 
simulation of fluid flows in aerodynamic configurations. 
These techniques are now' mature enough to be used rou- 
tinely. in conjunction with experimental results, in aerody- 
namic design. However, aerodynamic design optimization 
procedures that make efficient use of these advanced tech- 
niques are still being developed. 

The design of aircraft components, such as a wing, a 
fuselage or an engine, involves obtaining an optimal com- 
ponent shape that can deliver the desired level of component 
performance, subject to one or more constraints (such as 
maximum weight or cost) that the component(s) must sat- 
isfy. Aerodynamic design can be formulated as an optimi- 
zation problem that requires minimization of an objective 
function, subject to constraints. Many formal optimization 
methods have been developed and applied to aerodynamic 
design. These include inverse design methods, adjoint meth- 
ods, sensitivity derivative-based methods and traditional 
response surface methodology (RSM), 

Inverse design methods in aerodynamics are used to 
provide a component that responds in a pre-selected manner, 
for example, an aircraft wing that has a prescribed pressure 
distribution. The known inverse methods do not account for 
certain fluid parameters, such as viscosity, and are used in 
preliminary design only. 

Adjoint methods provide a designer with the gradient of 
the objective function. One advantage of this method is that 
the gradient information is obtained very quickly. However, 
where several technical disciplines are applied simulta- 
neously. it is often difficult to perform design optimization 
using this method; each discipline requires a different for- 
mulation. It is also difficult and expensive to quickly evalu- 
ate the effects of engineering tradeoffs, where the applicable 
constraints may be changed several times. It is also not 
possible to use existing experimental data or partial or 
unstructured data in the design process. 

A sensitivity derivative-based method typically requires 
that a multiplicity of solutions, w ith one parameter varied at 
a time, be obtained to compute a gradient of the objective 
function. The number of computations required grows lin- 
early with the number of design parameters considered for 
optimization, and this method quickly becomes computa- 
tionally expensive. This method is also sensitive to noise 
present in the design data sets. As with an adjoint method. 
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it is not possible to use existing experimental data or partial 
or unstructured data in the design process. 

RSM provides a framework for obtaining an optimal 
design, using statistical procedures, such as regression 
5 analysis and design of experiments. Traditional RSM uses 
low-degree regression polynomials in the relevant design 
variables to model the variation of an objective function. 
The polynomial model is then analyzed to obtain an optimal 
design. Several polynomial models may have to be con- 
io strutted to provide an adequate view of the design space. 
Addition of higher degree polynomials will increase the 
computational cost and will build in higher sensitivity to 
noise in the data used. 

Artificial neural networks (“neural nets’* herein) have 
15 been widely used in fields such as aerodynamic engineering, 
for modeling and analysis of flow control, estimation of 
aerodynamic coefficients, grid generation and data interpo- 
lation. Neural nets have been used in RSM-based design 
optimization, to replace or complement a polynomial-based 
20 regression analysis. Current applications of neural nets are 
limited to simple designs involving only a few design 
parameters. The number of data sets required for adequate 
modeling may increase geometrically or exponentially with 
the number of design parameters examined. A neural net 
25 analysis requires that the design space be populated with 
sufficiently dense simulation and/or experimental data. Use 
of sparse data may result in an inaccurate representation of 
the objective function in design space. On the other hand, 
inefficient use of design data in populating the design space 
30 can result in excessive simulation costs. Capacity control is 
critical to obtain good generalization capability. In some 
preceding work, this problem was alleviated by using a 
neural net to represent the functional behavior with respect 
to only those variables that result in complex, as opposed to 
35 simple, variations of the objective function; the functional 
behavior of the remaining variables was modeled using low 
degree polynomials. This requires a priori knowledge to 
partition the design variables into two sets. 

FIG. 1 graphically illustrates results of applying a simple 
40 NN analysis to a one -parameter model, namely, an approxi- 
mation to the second degree polynomial y=2*(0.5-xU at 
each of 3 pairs of training values (curve A) and at each of 5 
pairs of training values (curve B). Use of more than the 
minimum number (3) of training pairs clearly improves the 
45 fit over the domain of the variable x. It is theoretically 
possible that only Q+\ spaced apart training value pairs are 
needed to completely specify a Qth degree polynomial (for 
example, Q=6). However, because of the presence of noise, 
the theoretical minimum number of training value pairs is 
50 seldom sufficient to provide an acceptable lit. 

Use of neural network (NN) analysis of a physical object, 
in order to optimize response of the object in a specified 
physical environment, is well known. An example is opti- 
mization of a turbine blade shape, in two or three dimen- 
55 sions. in order to reproduce an idealized pressure distribu- 
tion along the blade surface, as disclosed by Rai and 
Madavan in “Aerodynamic Design Using Neural Net- 
works**, AIAA Jour., vol. 38 (2(X)0) pp. 173-182. NN 
analysis is suitable for multidimensional interpolation ol 
«> data that lack structure and provides a natural structure in 
which a succession of numerical solutions ol increasing 
complexity, or increasing fidelity to a real world environ- 
ment, can be represented and optimized. NN analysis is 
especially useful when multiple design objectives need to be 
met. 

A feed- forward neural net is a nonlinear estimation tech- 
nique. One difficulty associated with use ol a feed- forward 
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neural net arises from the need for nonlinear optimization to 
determine connection weights between input, intermediate 
and output variables. The training process can be very 
expensive when large amounts of data need to be modeled. 

In response to this, a support vector machine (SVM) 
approach, originally applied in statistical learning theory, 
has been developed and applied. Support vector machine 
analysis allows use of a feature space with a large dimen- 
sion, through use of a mapping from input space into feature 
space and use of a dual formulation of the governing 
equations and constraints. One advantage of an SVM 
approach is that the objective function (which is to be 
minimized to obtain the coefficients that deline the SVM 
model) is convex so that any local minimum is also a global 
minimum; this is not true for many neural net models. 
However, an underlying feature space (polynomial, Gauss- 
ian. etc. ) must be specified in a conventional SVM approach, 
and data resampling is required to implement model hybrid- 
ization. Hybridization is more naturally, and less expen- 
sively, applied in a neural net analysis. 

What is needed is a machine learning algorithm that 
combines the desirable features of NN analysis and of SVM 
analysis and does not require intimate a priori familiarity 
with operational details of the object to be optimized. 
Preferably, the method should automatically provide a char- 
acterization of many or all of the aspects in feature space 
needed for the analysis. 


FIG. 12 graphically illustrates data classification accord- 
ing to the invention. 

DESCRIPTION OF BEST MODES OF THE 
5 INVENTION 

Consider a feed-forward neural network 21 having an 

input layer with nodes 23-m (m=l 5), a hidden layer 

with nodes 25-// (n=l, 2. 3), and an output node 26, as 
10 illustrated schematically in FIG. 2. The first input layer node 
23-1 has a bias input value 1, in appropriate units. The 
remaining nodes of the input layer are used to enter selected 
parameter values as input variables, expressed as a vector 

p— (p i p A/ ), with 1. Each node 25-/? of the hidden 

15 layer is associated with a nonlinear activation function 



of a weighted sum of the parameter values p w/ , where C fmt is 
a connection weight, which can be positive, negative or zero, 
linking an input node 23-/// with a hidden layer node 25-/7. 
The output of the network 21 is assumed for simplicity, 
initially, to be a single-valued scalar. 


SUMMARY OF THE INVENTION 

The invention meets these needs by providing a hybrid of 
NN analysis and SVM analysis, referred to as NN/SVM 
analysis herein. In one embodiment, NN/SVM analysis 
begins with a group of associated, independent input space 
coordinates (parameter values), maps these coordinates into 35 
a feature space of appropriately higher dimension that 
includes a computed set of combinations (e.g., powers) of 
the input space coordinates with the assistance of the input 
and hidden layers of an NN, constructs an inner product 
formalism for the coordinates in feature space, obtains a 40 
solution to a minimization problem to compute Lagrange 
multiplier values that define the SVM, and returns to input 
space to complete a solution of the problem. 


FIG. 2 illustrates a conventional three-layer NN, with an 
input layer, a hidden layer and an output layer that receives 
and combines the resulting signals produced by the hidden 
layer. 

It is known that NN approximations of the format set forth 
in Eqs. (I) and (2) are dense in the space of continuous 
functions when the activation functions are continuous 
sigmoidal functions (monotonically increasing functions, 
with a selected lower limit, such as 0, and a selected upper 
limit, such as 1 ). Three commonly used sigmoidal functions 
are 


BRIEF DESCRIPTION OF THE DRAWINGS , 

FIG. 1 graphically illustrates an improvement in match of 
a polynomial, where an increased number of training pairs 
is included in a simple NN analysis. 

FIG. 2 is a schematic view of a three-layer feed-forward 
neural net in the prior art. 

FIG. 3 is a schematic view of a two-layer feed-forward 
NN/SVM system according to the invention. 

FIG. 4 is a flow chart of an overall procedure for prac- 
ticing the invention using an NN/SVM system. 

FIGS. 5, 6 and 7 graphically illustrate generalization 
curves obtained for a fifth degree polynomial, a logarithm 
function and an exponential function, respectively, using a 
hybrid NN/SVM analysis and 11 training values. 

FIGS. 8A/8B/8C are a flow chart for an RSM procedure 
used in practicing the invention. 

FIGS. 9A1-9C2 graphically illustrate evolution of an 
airfoil and corresponding pressure distribution obtained 
from an iterative NN/SVM analysis. 

FIGS. 10 and 11 A/1 IB illustrate data classification in two 
dimensions. 
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context of design optimization, a trained NN represents a 
response surface, and the NN output is the objective func- 
60 lion. In multiple objective optimization, different NNs can 
be used for di fie rent objective functions. A rapid training 
algorithm that determines the connection weights and 
coefficients D„ is also needed here. 

The approach set forth in the preceding does reasonably 
65 well in an interpolative mode, that is. in regions where data 
points (parameter value vectors) are reasonably plentiful. 
However, this approach rarely does well in an extrapolative 
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mode. In this latter situation, a precipitous drop in estimation 
accuracy may occur as one moves beyond the convex hull 
defined by the data point locations. In part, this is because 
the sigmoidal functions are not the most appropriate basis 
functions for most data modeling situations. Where the 
underlying function(s) is a polynomial in the parameter 
values, a more appropriate set of basis functions is a set of 
Legendre functions < it* the parameter value domain is finite), 
or a set of Laguerrc or Hermite functions (if the parameter 
value domain is infinite). Where the underlying function(s) 
is periodic in a parameter value, a Fourier series may be 
more appropriate to represent the variation of the function 
with that parameter. 

Two well known approaches are available for reducing 
die disparity between an underlying function and an activa- 
tion function. A first approach, relies on neural nets and uses 
appropriate functions of the primary variables as additional 
input signals for the input nodes. These functions simplify 
relationships between neural net input and output variables 
but require a priori knowledge of these relationships, includ- 
ing specification of all the important nonlinear terms in the 
variables. For example, a function of the (independent) 
parameter values x and y. such as 

M.v.v -,v 2 +h .x- v+< ■ r’+J-.v+t'* \+J. ( 5 ) 

where a, b, e, d, e and f are constant coeflieients, would be 
better approximated if the terms x, y, x 2 , x y and y 2 are all 
supplied to the input nodes of the network 21. However, in 
a more general setting with many parameters, this leads to 
a very large number of input nodes and as-yet-undetermined 
connection weights C,„„. 

A second approach, referred to as a support vector 
machine (SVM). provides a nonlinear transformation from 
the input space variables p,„ into a feature space that contains 
the original variables p„, and the important nonlinear com- 
binations of such terms (e.g., (p,) 2 . (p')(p 2 )\pA/) 2 and exp 
(p 2 )) as coordinates. For the example function h(p,,p 2 ) set 
forth in Eq. (5), the five appropriate feature space coordi- 
nates would be p,. p 2 , (p,) 2 , prp^ and (p 2 )~. Very high 
dimensional feature spaces can be handled efficiently using 
kernel functions for certain choices of feature space coor- 
dinates. The total mapping between the input space of 
individual variables (first power of each parameter p,„) and 
the output space is a hyperplane in feature space. For a 
model that requires only linear terms and polynomial terms 
of total degree 2 (as in Eq. (5)), in the input space variables, 
the model can be constructed efficiently using kernel func- 
tions that can be used to define inner products between 
vectors in feature space. However, use of an SVM requires 
a priori knowledge of the f unctional relationships between 
input and output variables. 

The mapping between the input space parameters and the 
output function is defined using a kernel function and certain 
Lagrange multipliers. The Lagrange multipliers are obtained 
by maximizing a function that is quadratic and convex in the 
multipliers, the advantage being that every local minimum is 
also a global minimum. By contrast, a neural net often 
exhibits numerous local minima of the training error(s) that 
may not be global minima. However, several of these local 
minima may provide acceptable training errors. The result- 
ing multiplicity of acceptable weight vectors can be used to 
provide superior network generalization, using a process 
known as network hybridization. A hybrid network can be 
constructed from the individual trained networks, without 
requiring data re-sampling or similar expensive techniques. 
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An attractive feature of a neural net. vis-a-vis an SVM, is 
that the coordinates used in a feature space do not have to be 
specified (e.g.. via kernel functions). However, use of an 
SVM. in contrast to use of a neural net. allows one to 
s introduce features spaces with a large number of dimen- 
sions, without a corresponding increase in the number of 
coefficients. 

A primary contribution of the present invention is to 
provide a mechanism, within the NN component, for deter- 
mining at least the coordinate (parameter) combinations 
needed to adequately define the feature space for an SVM, 
without requiring detailed knowledge of the relationships 
between input parameters and the output function. 

! 5 FIG. 3 is a schematic view of an NN/SVM system 31. 

including an NN component and an SVM component, 
according to the invention. The system 31 includes input 

layer nodes 33-/ (i=l 5) and hidden layer nodes 35-/ 

(j=l, 2. 3 ). FIG. 3 also indicates some of the connection 
-° weights associated with connections of the input layer 
terminals and the hidden layer terminals. More than one 
hidden layer can be provided. The hidden layer output 
signals are individually received at an SVM 37 for further 
processing, including computation of a training error. If the 
computed training error is too large, one or more of the 
connection weights is changed, and the (changed) connec- 
tion weights are returned to (he NN component input ter- 
minals for repetition oflhe procedure. Optionally, the SVM 
37 receives one or more user-specified augmented inner 
- ,() product or kernel prescriptions (discussed in the following), 
including selected combinations of coordinates to be added, 
from an augmentation source 38. 

FIG. 4 is a flow chart illustrating an overall procedure 
according to the invention. In step 41, the system provides 
^ (initial) values for connection weights C, m/ for the input 
layer-hidden layer connections. These weights may be ran- 
domly chosen. The input signals may be a vector of param- 
eter values p= (p, p A/ ) (M=5 in FIG. 3) in parameter 

space. In step 42. output signals from the hidden layer are 
40 computed to define the feature space for the SVM. The NN 
component of the system will provide appropriate combi- 
nations of the parameter space coordinates as new coordi- 
nates in a feature space for the SVM (e.g., u,=p,. u 2 =p 2 . 
uy=p, 2 , u 4 =p,-p 2 , u s =p 2 2 , from Eq. (5)) 

In step 43, feature space inner products that are required 
for the SVM are computed. In step 43 A, user-specified 
feature space coordinates and corresponding inner products 
and kernel functions are provided. Note that the feature 
space is a vector space with a corresponding inner product. 

In step 44, a Lagrange functional is defined and mini- 
mized, subject to constraints, to obtain Lagrange multiplier 
values for the SVM. See the Appendix for a discussion of a 
Lagrange functional and associated constraints. In step 45. 
55 the NN connection weights and the Lagrange multiplier 
coefficients are incorporated and used to compute a training 
error associated with this choice of values within the 
NN/SVM. 

In step 46. the system determines if the training error is no 
W) greater than a specified threshold level. If the answer to the 
query in step 46 is “no", the system changes at least one 
connection weight, in step 47. preferably in a direction that 
is likely to reduce the training error, and repeats steps 42^46. 
If the answer to the query in step 46 is “yes . the system 
65 interprets the present set of connection weights and 
Lagrange multiplier values as an optimal solution of the 
problem, in step 48. 
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Note that steps 42-48 can be embedded in an optimization 
loop, wherein the connection weights are changed according 
to the rules of the particular optimization method used. 

The hybrid NN/SVM system relies on the following 
broadly staled actions: ( I ) provide initial random (or other- 
wise specified) connection weights for the NN: (2) use the 
activation function(s) and the connection weights associated 
with each hidden layer unit to construct inner products for 
the SVM: (3) use the inner products to compute the 
Lagrange multiplier values; (4) compute a training error 
associated with the present values of the connection weights 
and Lagrange multiplier values; (5) if the training error is too 
large, change at least one connection weight and repeal steps 
(2)-(4); (6) if the training error is not too large, accept the 
resulting values of the connection weights and the Lagrange 
multiplier values as optimal. 

This method has several advantages over a conventional 
SVM approach. First, coordinates that must be specified a 
priori in the feature space for a conventional SVM are 
determined by the NN component in an NN/SVM system. 
The feature space coordinates are generated by the NN 
component to correspond to the data at hand. In other words, 
the feature space provided by the NN component evolves to 
match or correspond to the data. A feature space that evolves 
in this manner is referred to as * v data-adaptive." The feature 
space coordinates generated by the NN component can be 
easily augmented with additional user-specified feature 
space coordinates (parameter combinations) and kernel 
functions. 

Second, use of activation functions that are nonlinear 
functions of the connection weights in the NN component 
reintroduces the possibility of multiple local minima and 
provides a possibility of hybridization without requiring data 
resampling. 

The feature spaces generated by the NN hidden layer can 
be easily augmented with high-dimensional feature spaces 
without requiring a corresponding increase in the number of 
connection weights. For example, a polynomial kernel con- 
taining all monomials and binomials (degrees one and two) 
in the parameter space coordinates can be added to an inner 
product generated by the SVM component, without requir- 
ing any additional connection weights or Lagrange multi- 
plier coefficients. 

The NN/SVM system employs nonlinear optimization 
methods to obtain acceptable connection weights, but the 
weight vectors thus found are not necessarily unique. Many 
di tie rent weight vectors may provide acceptably low train- 
ing errors for a given set of training data. This multiplicity 
of acceptable weight vectors can be used to advantage. If 
validation data are available, one can select the connection 
weight vector and resulting NN/SVM system with the 
smallest validation error. In aerodynamics, this requires 
additional simulations that can he computationally expen- 
sive. 

If validation data are not available, multiple trained NNs 
or NN/SVM systems can be utilized by creating a hybrid 
NN/SVM. A. weighted average of N output signals from 
trained NN/SVMs in a hybrid NN/SVM is formed as a new 
solution. Where the weights are equal, if errors for the N 
individual output solutions are uncorrelated and individually 
have zero mean, the least squares error of this new solution 
is approximately a factor of N less than the average of the 
least squares errors for the N individual solutions. When the 
errors for the N individual output solutions are partly 
correlated, the hybrid solution continues to produce a least 
squares error that is smaller than the average of the least 
squares errors for the N individual solutions, but the differ- 
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once is not as large. The N trained NN/SVMs used to form 
a hybrid system need not have the same architecture or he 
trained using the same training set. 

FIG. 5 graphically illustrates results of applying an 
5 NN/SVM analysis according to the invention to a six- 
parameter model, namely, an approximation to the fifth 
degree polynomial y=x( l-x 2 )(4~x : ). Data are provided at 
each of II training locations (indicated by small circles on 
the curve) in the domain of the variable x. After a few 
io iterations of an NN/SVM analysis, the 11 training values, 
(x^.y A )=(x A ,x A ( l-x A 2 )(4-x A 2 )), provide the solid curve as a 
generalization, using the NN/SVM analysis. The dashed 
curve (barely visible in FIG. 5) is a plot of the original fifth 
order polynomial. 

15 FIG. 6 graphically illustrates similar results of an appli- 
cation of the NN/SVM analysis to a logarithm function, 
y=ln(x+4). using 1 1 training values. The solid curve is the 
generalization provided by the NN/SVM analysis. 

FIG. 7 graphically illustrates similar results of an appli- 
20 cation of the NN/SVM analysis to an exponential function, 
y=6‘exp(-0.5*x :! ), using 1 1 training values. The solid curve 
is the generalization provided by the NN/SVM analysis, 
using the 1 1 training values. 

The generalization in each of FIGS. 5, 6 and 7 is vastly 
25 superior to corresponding generalizations provided by con- 
ventional approaches. In obtaining such a generalization, the 
same computer code can be used, with no change of param- 
eters or other variables required. 

FIGS. 8 A, 8B and 8C are a flow chart illustrating the 
30 application of a response surface methodology (RSM) used 
in this invention to obtain an optimal cross-sectional shape 
of an airfoil, as an example, where specified pressure values 
at selected locations on the airfoil perimeter are to be 
matched as closely as possible. In step 81, a set of param- 

35 eters, expressed here as a vector p=(p, p A/ ), is provided 

that adequately describes the airfoil cross-sectional shape 
(referred to as a “shape" herein), where M ( = 1 ) is a selected 
positive integer. For example, the airfoil shape might be 
described by (1) first and second radii that approximate the 
40 shape of the airfoil at the leading edge and at the trailing 
edge, (2) four coefficients that describe a tension spline fit of 
the upper perimeter of the airfoil between the leading and 
trailing edge shapes, and (3) four coefficients that describe a 
tension spline fit of the lower perimeter of the airfoil 
45 between the leading and trailing edge shapes, a total of ten 
parameters. In a more general setting, the number M of 
parameters may range from 2 to 20 or more. 

In step 82, initial values of the parameters, p=p0, are 
provided from an initial approximation to the desired airfoil 
50 shape. 

In step 83, optimal data values P(rp.opt) (e.g., airfoil 
pressure values or airfoil heat transfer values) are provided 

at selected locations r A =(x A .,y A ,z A ) (k= 1 K) on the airfoil 

perimeter. 

55 In step 84, an equilateral M -simplex, denoted MS(p0), is 
constructed, with a centroid or other selected central location 
at p—p(), in M -dimensional parameter space, with vertices 
lying on a unit radius sphere. Each of the M+l vertices of the 
M -simplex MS(p0) is connected to the centroid. p=p(), by a 

oo vector Ap(m) (m=l M+l) in parameter space. More 

than the M+l vertices can be selected and used within the 
M -simplex. For example, midpoints of each of the 
M(M+I )/2 simplex edges can be added to the M+l vertices. 
These additional locations will provide a more accurate 
65 NN/SVM model. 

In step 85, a computational fluid dynamics (CFD) or other 
calculation is performed for an extended parameter value 
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scl. consisting of the parameter value vectors p=p() and each 
of the M+l M-simplex vertices. P=P r>) =P0+Ap(m), to 
obtain a calculated pressure distribution P(r A ;p, .«.,.,) at each of 
the selected perimeter locations, r=r A for each of these 
parameter value sets. One hybrid NN/SVM is assigned to 
perform the analysis for alt vertices in the M-simplex 
MS(pO) at each location r*. That is, a total of K NN/SVM 
systems are used to model the overall pressure dependence 
on the parameters p,„. The calculated pressure distribution 
P(r A ;p ir i.,) and/or the airfoil can be replaced by any other 
suitable physical model, in aerodynamics or in any other 
technical field or discipline. Used together, the trained 
NN/SVM systems will provide the pressure distribution 
P(r 4 ;p) for general parameter value vectors p. 

In step 86, a first objective function, such as 


K 

OliJifK p(K I ) = ^ «■* {/‘(a ; />) - /V< : opt)] 2 . 

< = i 


(6 A I 


is introduced, where {w A } is a selected set of non-negative 
weight coefficients. 

In step 87, the minimum value of the first objective 
function OBJ(p;p():l ) and a corresponding parameter vector 
p=p(min) are determined for parameter vectors p within a 
selected sphere having a selected diameter or dilatation 
factor d, defined by lp-p()l^d, with l<d^ 10. The process is 
performed using a nonlinear optimization method. Other 
measures of extrapolation can also be used here. 

In step 88, the system calculates a second objective 
function, which may be the first objective function or 
(preferably) may be defined as 


i , (6B) 

OliJip: p()\ 2) = \Pir t :p: CFD) - F(r k :opt)}~, 

k = i 


where P(r*;p;CFD) is a pressure value computed using a 
CFD simulation, for p= p(min) and p=p(). The system then 
determines if OBJ(p(min);pO;2)<OBJ(pO;p();2) for the inter- 
mediate minimum value parameter vector. p=p(min). One 
can use the first objective function OBJ(p;p(); 1 ), defined in 
Eq. (6A), rather than the objective function OBJ(p;p();2) 
defined in Eq. (6B). for this comparison, but the resulting 
inaccuracies may be large. 

If the answer to the query in step 88 is “no" for the choice 
of dilatation factor d. the dilatation factor d is reduced to a 
smaller value d' ( l<d’<d). in step 89, and steps 88 and 89 are 
repeated until the approximation pressure values {P(r A ,p)} A 
for the extrapolated parameter value set provide an 
improved approximation for the optimal values for the same 
airfoil perimeter locations. r=r*. 

If the answer to the query in step 88 is "yes", the system 
moves to step 90. uses the (modified) objective function and 
uses the intermediate minimum-cost parameter value set. 
p=p(min). which may lie inside or outside the M-simplex 
MS(pO) in parameter space. Minimization of the objective 
function OBJ(p;pO) may include one or more constraints, 
which may be enforced using the well known method of 
penally functions. The (modified) objective function defini- 
tion in Eq. (6A) (or in Eq. (6B)) can be replaced by any other 
positive definite definition of an objective function, for 
example, by 


K 


5 



<60 


where q is a selected positive number. 

If the original parameter value set p has an insufficient 
10 number of parameters, this will become evident in the 
preceding calculations, and the (modilied) objective func- 
tion OBJ(p(min);pO) or ()BJ(p(min);p()) :: will not lend 
toward acceptably small numbers. In this situation, at least 
one additional parameter would be added to the parameter 
|S value set p and the procedure would be repealed. In effect, 
an NN/SVM procedure used in an RSM analysis w ill require 
addition of (one or more) parameters until the convergence 
toward a minimum value that is acceptable for an optimized 
design. 

20 In step 91, the system determines if the (modilied) objec- 
tive function OBJ(p(min);p()) ::: is no greater than a selected 
threshold number (e.g., 1 or l(r 4 , in appropriate units). If the 
answer to the query in step 91 is “no", a new M-simplex 
MSfp'O) is formulated, in step 92, with p'()=p(iiiin) as the 
25 new' center, and steps 85-90 are repeated at least once. Each 
time, a new parameter value set, p=p(min). is determined 
that approximately minimizes the objective function OBJ(p; 
p'O). 

If the answer to the query in step 91 is “yes", the system 
interprets the resulting parameter set. p=p(min), and the 
design described by this parameter set as optimal, in step 93. 
The method set forth in steps 81-93 is referred to herein as 
a response surface method. 

FIGS. 9A1-9C2 illustrate a sequence of partly-optimized 
designs for an airfoil, obtained using the invention, and 
compare each such design shape and corresponding airfoil 
pressure distribution to an target airfoil design shape and 
corresponding target airfoil pressure distribution. The objec- 
40 live function is defined as mean square error between 
resulting and target pressure distribution at a sequence of 
selected locations on the airfoil perimeter. One begins in 
FIG. 9A1 with a curvilinear shape of approximately uniform 
thickness, which provides a pressure distribution p along the 
45 airfoil perimeter as illustrated graphically in FIG. 9A2. 
FIGS. 9B1 and 9C1 illustrate the results of second and 
fourth iterative applications of an NN/SVM analysis accord- 
ing to the invention, and FIGS. 9B2 and 9C2 graphically 
illustrate the pressure distributions corresponding to FIGS. 
50 9B1 and 9C1, respectively. Each iteration brings the result- 
ing airfoil shape and pressure distribution closer to the target 
shape and target pressure distribution. Alter a fourth itera- 
tion of the NN/SVM analysis, the airfoil shape, shown in 
FIG. 90, produces a pressure distribution, shown in FIG. 
55 9C2, that nearly precisely matches the target airfoil pressure 
distribution. Computations for this iterative sequence 
required about 8 minutes on a 16-processor SGI Origin 
computer. 

In a second embodiment. NN/SVM analysis is applied to 
60 data classification in a multi-dimensional vector space. In 
data classification, a discrimination mechanism must be 
determined that divides the data points into (at least) a lirst 
set of data points that satisfy a selected criterion, and a 
second set of data points that either do not satisfy the (first) 
65 criterion or that satisfy an inconsistent second criterion. FIG. 
10 illustrates a collection of first set data points (“x") and 
second set data points (“o") in two (parameter) dimensions 
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thal arc easily separated by a linear function of the two 
parameter coordinates, namely 

f\(x.y)=<i\+hy-t=(). ( 7 ) 

where a. b and c are selected real values, with at least one 
of a and b being non-zero: All data points in the first data set 
and in the second data set lie on opposite sides of the line 
(hyperplane) lj(x,y)=(). Here, the data point separation is 
straightforward. 

FIGS. 11 A and 11 B illustrate a collection of first set data 
points (**x“) and second set data points (“o") that cannot be 
separated using a linear function of the two coordinates. An 
appropriate separation function may be 

l 2 (\. ) 2 ±UL.x+r-\-t > f =1 . (8 ) 

where a d+b-e=() and a, b, e, d, e and g are selected real 
values, not all zero. The choice of the plus (+) sign in Eq. (8) 
produces an ellipse, and the choice of a minus (-) sign in Eq. 
(8) produces a hyperbola. In this instance, one set of 
appropriate coordinates for hyperplane separation in feature 


space is 

u i =x. <9A) 

u 2 =y, (9B) 

Ut=(a\+by-i) 2 . (90 

u 4 ={dx+t'y~i>) 2 . (9D) 

in which the separating hyperplane in feature space becomes 

h,±m 4 - 1=0. (10) 


The power of an SVM resides, in part, in its use of a qth 
order polynomial kernel (as an example) for vectors a and 
p, such as 

*:«x,p)=<a-p+] >", (in 

where q is a selected positive integer (e.g., q=2), rather than 
requiring an a priori definition of the polynomial terms to be 
used, as in Eqs. (9A)-(9D). 

An advantage of the present invention, using NN/SVM 
analysis, over a conventional SVM analysis is that the 
kernel, such as the one given in Eq. ( 1 1 ), and the associated 
feature space need not be specified a priori; the appropriate 
feature space is automatically generated by the NN compo- 
nent of the NN/SVM system during the training process. 

FIG. 12 illustrates an application of the NN/SVM system 
to data classification, with M=2. Two classes of data that are 
separable, indicated as crosses and squares, are provided for 
the system. The exact boundary between the two classes is 
defined by first and second intersecting ellipses in two 
dimensions, with the major axes being oriented at 45° and at 
135" relative to an x-axis in an (x,y) region p defined by 

p= ! (.v.y WOS.vg 2.5,0£yS 2.5 } . (12) 

Four hundred data points were randomly generated in this 
region and were first classified according to the exact 
boundaries. The boundaries were then removed, and only 
the locations of the data points were provided to the 
NN/SVM system. The resulting decision boundary gener- 
ated by the NN/SVM system is shown as a solid line in FIG. 
12. More generally, if M-parameter data points are provided, 
with M^2, the data separation surface or hyperplane will 
have dimension at most M-I. 

The NN/SVM system provides a perfect classification of 
the original data, with zero mis-assignments. without requir- 


12 

ing any specification of kernel functions or feature spaces. 
Where the solid boundary line and the dotted boundary lines 
diner, no data points were located in the intervening regions 
between these boundaries. Provision of additional data 
points in one or more of these intervening regions would 
provide a resulting (solid) NN/SVM boundary line thal is 
closer to the exact (dotted) boundary line. 

If r is a ratio of the sum of the absolute value of the 
m intervening regions corresponding to the boundary lines 
mismatch, and the area of the square (6.25 units 2 in FIG. 12), 
the ratio r is a very small number that will lend toward zero 
as the number of data points (assumed to he approximately 
uniformly distributed) increases without hound. Addition- 
K ally, r (defined as a percentage) represents the number of 
misclassi locations (also expressed as a percentage) that an 
NN/SVM-generatcd boundary will produce on a very large 
test set. 

20 

APPENDIX 

Examples of an NN analysis and of an SVM analysis are 
presented here. The invention is not limited to a particular 
~ NN analysis or to a particular SVM analysis. 

Consider an object, represented by a group of coordinates 

x=(x‘, x 2 x"), for which some physical feature or 

response of the object is to be optimized. The object may be 
30 a aircraft wing or turbine blade for which an ideal pressure 
distribution at specified locations on the object is to be 
achieved as closely as possible. The object may be a 
chemically reacting system with desired percentages of linal 
compounds, for which total thermal energy output is mini- 
35 mized. The object may be represented at spaced apart 
locations or at spaced apart times by a group of independent 
coordinates, and an objective or cost function is presented, 
representing the response to be optimized. One or more 
constraints, either physical or numerical, are also set down, 
40 if desired. 

In an NN analysis, one relevant problem is minimizing 
empirical risk over a sum of linear indicator or characteristic 
functions 
45 



50 

where 0 is an indicator or characteristic function, x is a 
coordinate vector and w is a vector of selected weight 
coefficients. Consider a training set of (N+I )-tuples (Xj.y,). 

55 (x 2 ,y 2 ) (x A -,y*), where each x=(x/, x ; 2 x A) is an 

N -tuple representing a vector and y / is a scalar having only 
(he values -I or +1 . 

The indicator function 0(z) has only two values, 0 and 1, 
and is not generally differentiable with respect to a variable 

6° in its argument. The indicator function 0(z) in Eq. (A-l) is 
often replaced by a general sigmoid function S(z) that is 
differentiable with respect to z everywhere on the finite real 
line, is monolonically increasing with z. and satisfies 

65 I Jin , . .S’(-)=0. ( A-2a) 


Uni ^ 5( ~)=l . 


( A-2b) 
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Examples of suitable sigmoid functions include the follow- 
ing: 

.V< -)=!/• |+cxpi HX-)j. 


l+t;mhl|i--+X>|/2 


ir+2-t;m 1 1 i>- \/2k. 

where a, p and 5 are selected positive values. The indicator 
sum f(x,w) in Eq. ( A- 1 ) is replaced by a modified sigmoid 
sum 


(ii.W U') - 



(A- 3) 


where S is a selected linear or nonlinear function. 

In order to minimize the empirical risk, one must deter- 
mine the parameter values w, that minimize an empirical risk 
functional 


A 

= £<y, - Hv,. U)) : / K. 


(A-4) 


which is dilferentiable in the vector components w. One 
may. for example, use a gradient search approach to mini- 
mize R,. m/ ,(w). The search may converge to a local mini- 
mum, which may or may not be a global minimum for the 
empirical risk. 

Assume, lirst. that the training data {(x^y,)} can be 
separated by an optima! separating hyperplane, defined by 

(vr-.v, >-£=(), (A-5) 


where g partly delines the hyperplane. A separating hyper- 
plane satisfies 

<iv.v y >-£= 1 (v,= h. ( A-fia) 

(v,=-l). (A-6b> 

An optimal separating hyperplane maximizes the functional 
Ol w)=(ivu)/2. < A-7 > 


with respect to the vector values w and the value g, subject 
to the constraints in Eqs. (A-6aMA-6b). Unless indicated 
otherwise, all sums in the following are understood to be 
over the index j (j=l K). 

A solution to this optimization problem is given by a 
saddle point of a Lagrange functional 


A 

/.Or, }•* <0 = Of - ir)/2 - ■ »r) - ,*»)■< y, - 1 1}. 


(A-S) 


At a saddle point, the solutions (w.g.a) satisfy the relations 

f)l./rig=0. (A-O) 

<)l./()w=0, (A- 10) 


with the associated constraint 


uj=0. 


(A- ID 
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Equation (A-9) yields the constraint 


V'. v=0. 


(A-12) 


Equation (A- 10) provides an expression for the parameter 
1() vector w of an optimal hyperplane as a linear combination 
of vectors in the training set 

ir=S\y« ; \V,. (A-13) 

An optimal solution (w.g.a) must satisfy a Kuhn-Tueker 
is condition 

tx/'ju v, ir)-y»(y,-l 1=0 ( = 1 A). (A-14) 

Only some of the training vectors, referred to herein as 
“support vectors." have non-zero coefficients in the expan- 
20 sion of the optimal solution vector w. More precisely, the 
expansion in Eq. (A- 1 3) can be rewritten as 

u’=I\y(X / -.v / . (A- 15) 

support vectors 

25 

Substituting the optimal vector w back into Eq. ( A-S ) and 
taking into account the Kuhn-Tueker condition, the 
Lagrange functional to be minimized is re-expressed as 

30 

^ * ( A- 16) 

Utt) = \ <r t 1 n >‘*i ■ y # 'Yj *(.v, -.v,). 


- This functional is to be maximized, subject to the constraints 
expressed in Eqs. (A- 13) and (A-14). Substituting the 
expression for optimal parameter vector w into Eq. (A-14), 
one obtains 

(vr-.v)-x'=lix ; -( \yv)-v=0. (A- 1 7) 

The preceding development assumes that the training set 
data {(x^y,)} are separable by a hyperplane. If these data are 
not separable by a hyperplane, one introduces non-negative 
slack variables x,(j=l K) and a modified functional 

xJjOvMm’-ivH-OLx,. (A* 18) 

subject to the constraints 

y / .((u-.v / l- i ">= l-t ; tA-19) 

50 

where the (positive) coefficient C corresponds to an inter- 
penetration of two or more groups of training set (N+1)- 
tuples into each other (thus, precluding separation by a 
hyperplane). Repeating the preceding analysis, where the 
55 functional O(w) replaces the term(w w). an optimal solution 
(w.g.a) is found as before by maximizing a quadratic form, 
subject to the modified constraints 

lu,y=0.. ( A -20a) 

60 ()=e<X,==C\ ( A-20bi 

Use of (only) hyperplanes in an input space is insufficient lor 
certain classes of data. See the examples in FIGS. 11 A and 

11B. 

65 In a support vector machine, input vectors are mapped 
into a high dimension feature space Z through a selected 
nonlinear mapping. In the space Z, an optimal separating 
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hyperplane is constructed that maximizes a certain A-margin 
associated with hyperplane separation. 

Firsts consider a mapping that allows one to construct 
decision polynomials of degree 2 in the input space. One 
creates a (quadratic) feature space Z having dimension 
M=N(N+3)/2, with coordinates 


iij=\'{j= I N: N coordi miles) (A-2l;i) 

a, t ,v= V t/ = 1 N : N coordinates ) ( A-2 1 b ) 

i V:\-Vr-. \ v , v v - (N(N-l)/2 coordi- 
nates). ( A-2 Ic) 


A separating hyperplane constructed in the space Z is 
assumed to be a second degree polynomial in the input space 
coordinates x^j-1, . . . N). 

By analogy, in order to construct a polynomial of degree 
k in the input coordinates, one must construct a space Z 
having of the order of N* coordinates, where one constructs 
an optimal separating hyperplane. For example, for k=4. the 
maximum number of coordinates needed in the space Z is 


+ A| (A-22) 


which is about I0 8 coordinates for a modest size input space 
of N=100 independent coordinates. 

For a quadratic feature space Z, one First determines a 
kernel function K of inner products according to 

(«, i‘U,.)=K(x n .x r )=K(x r ,x n ) (El, L2-\ N{N+?>)f 

2). (A-22) 

One constructs nonlinear decision functions 

/(.v )-s\>n { "La.jKix.Xj H-W) } ( A-24 ) 

support vectors 

that arc equivalent to the decision function O(x) in Eq. 
{A- 18). By analogy with the preceding, the coefficients a ; 
are estimated by solving the equation 

WUx Vi Ha-a/.v-Ay K(x,,Xj) < A- 25 ) 

with the following constraint (or sequence of constraints) 
imposed: 

I(X /V =0 (A-26a) 

a j^i). (A- 26b) 

Mercer ( 1 909) has proved that a one-to-one correspon- 
dence exists between the set of symmetric, positive definite 
functions K(x,y) defined on the real line that satisfy 

J J K(,v. v ) ./( .v ) fix ) d\ iiy § 0 < A-27 ) 

for any L2-inlegrable function f(x) satisfying 

J fix) 2 d\ <°o (A‘28) 

and the set of inner products defined on that function space 
{f}. Thus, any kernel function K(x /l .x /2 ) satisfying condi- 
tions of the Mercer theorem can be used to construct an inner 
product of the type set forth in Eq. (A-23). Using different 
expressions for the kernel K(x / ,,x /2 ), one can construct 
di Here nt learning machines with corresponding nonlinear 
decision functions. 


For example, the kernel function 

A'(.v ',.y" )= [ (.v'-.v" )+ 1 j q. ( A-2 1 ) ) 

can be used to specify polynomials of degree up to q 
5 (preferably an integer). 

Much of the preceding development is taken from V.N. 
Vapnik, “An Overview of Statistical Learning Theory". 
IEEE Trans. Neural Networks, vol. 10 (1999). pp. 988-999. 
The present invention provides a hybrid approach in which 
10 the input layer and hidden layer(s) of an NN component are 
used to create a dala-adaplive feature space for an SVM 
component. As indicated in the preceding, the combined 
NN/SVM analysis of the invention is not limited to the 
particular NN analysis or to the particular SVM analysis set 
15 forth in this Appendix. 

What is claimed is: 

1. A computer implemented machine learning method for 
use in engineering applications, including but not limited to 
optimizing designs, classifying data and generating regres- 
sion estimates, that is a hybrid of neural net (“NN”) analysis 
and support vector machine (“SVM") analysis, the method 
comprising: 

(a) providing an NN component, having an input layer 
and a hidden layer and an input vector space, where the 
NN component automatically generates coordinates in 
a feature vector space, and providing an SVM compo- 
nent that utilizes the feature vector space: 

(b) selecting a group of parameters and combinations of 
3<) parameters and providing a feature space coordinate, in 

the feature vector space, for each selected parameter 
and selected parameter combination in the input space 
for use in at least one of optimizing a design, control- 
ling a physical or chemical process, classifying data 
35 and generating regression estimates for a collection of 
the data: 

(c) providing at least one vector of candidate parameter 
values for each of the group of parameters in the input 
space; 

40 (d) providing initial values for connection weights 

between the input layer and the hidden layer for the NN 
component; 

(e) computing hidden layer output signals, corresponding 
to the connection weight values, for each of the param- 

45 eter value vectors; 

(f) using at least one hidden layer output signal as a 
feature space coordinate for the SVM component; 

(g) determining inner product values of a selected number 

of at least two feature space coordinates; 

50 

(h) providing a Lagrange functional using the determined 
inner product values; 

(i) providing at least two constraints, expressed in terms 
of Lagrange multipliers and input vector space data; 

55 (j) minimizing the Lagrange functional, subject to at least 

one selected constraint, to obtain Lagrange multiplier 
values corresponding to the minimized Lagrange func- 
tional; 

(k) computing a training error, using the connection 
60 weights for the NN component and the Lagrange 

multiplier values for the SVM component; 

(l) when the computed training error is greater than a 
selected threshold value, changing at least one of the 
connection weights and repeating steps (e)-(k) at least 

65 once, wherein at least one feature space coordinate 
value changes automatically in response to change in 
the at least one connection weight; and 
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(m) when the computed training error is not greater than 
the threshold value, interpreting the NN component 
with the associated connection weights and the SVM 
component with the associated Lagrange multipliers as 
a trained NN/SVM system. 5 

2. The method of claim 1. further comprising augmenting 
said inner product value with at least one user-speeilied 
inner product value in said SVM component. 

3. The method of claim 1. further comprising: 

providing a collection of N data points in an M -dimen- 
sional space for said input space, where M = 2 and 
N^2. and w here each data point is assigned an indi- 
cium associated with one of at least first and second 
mutually exclusive sets; and 15 

applying the method of claim 1 for determination of a 
separation surface in the M -dimensional space that 
separates the data points into at least first and second 
mutually exclusive regions that contain substantially all 
data points in the first set and in the second set, 20 
respectively. 

4. The method of claim 3, further comprising providing a 

visually perceptible view of at least a portion of said 
separation surface in at least two dimensions. ^ 

5. The method of claim 1, wherein said use in engineering 
applications comprises design of a airfoil representing a 
wing or other control surface of an aircraft. 

6. The method of claim 5, further comprising: 

providing an optimization method: and 30 

using the optimization method in at least one of steps (e) 
through (k) to obtain at least one of said connection 
weight values, said Lagrange multiplier values and said 
inner product value for said aircraft airfoil. 35 

7. The method of claim 5. further comprising determining 
an optimized design for said aircraft airfoil by applying a 
response surface analysis to said design, using said trained 
NN/SVM system. 

8. The method of claim 7, further comprising providing a 40 
selected optimization procedure in determining said opti- 
mized design. 
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9. The method of claim 5. further comprising computing 
said training error by a process comprising: 

providing a collection of at least two desired pressure 
values, with each desired pressure value corresponding 
to a location on said airfoil: 

computing a pressure value at each of the selected loca- 
tions on said airfoil, using said minimized Lagrangian 
functional: and 

computing said training error as a sum of magnitudes ol 
differences between the desired pressure value and the 
computed pressure value at each of the selected loca- 
tions on said airfoil. 

10. The method of claim 5. further comprising computing 
said airfoil by a process comprising: 

(n ) providing a collection of a least two desired pressure 
values, with each desired pressure value corresponding 
to a location on said airfoil: 

(o) providing an initial airfoil shape: 

( p ) computing a pressure distribution for said airfoil for at 
least one perturbation in the initial airfoil shape: 

(q) representing variation of pressure with the at least one 
perturbation in the initial airfoil shape, using said NN 
component and said SVM component, wherein said 
NN input vector corresponds to the at least one pertur- 
bation in the initial airfoil shape and an SVM output 
signal corresponds to at least one change in the pressure 
distribution: 

(r) computing an objective function value, which is to be 
minimized for said airfoil design optimization, as a sum 
of magnitudes of a power of differences between the 
desired pressure value and a pressure value using said 
NN component and said SVM component: 

(s) when the objective function value is greater than a 
selected threshold value, repeating steps (pHr) at least 
once: and 

(l) when the objective function not greater than the 
threshold value, interpreting this condition as indicat- 
ing that an optimal airfoil design is identilied. 

11 . The method of claim 1 , wherein said use in engineer- 
ing applications comprises design of a airfoil representing at 
least one turbine or compressor airfoil. 

* * 
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